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Variational methods 


Interested in P(x) = 790) = zer. 


E(x) is simple, but not simple enough. 

Idea Approximate Р(х) by а simpler distribution Q(x; 0). 
Adjust 0 to get the ‘best’ approximation. 
Then approximate HP) by Loe )Q(x; 0") 


How to measure 'best? Possible ideas: 
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Distances between probability distributions 
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and Q(x) = (1/4, 1/4, 1/4,1/4), 


which is bigger: 
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iie function: Variational free energy 
p? Q(x:0)E o> Q(x: 0) ng = 


Е(0) is lower-bounded by — log Z 
F(0) = Dix(Q||P) - log Z 
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Examples 


Inferring mu and sigma 


Two coupled spins 
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Variational methods - example er 


Approximate the spin system whose energy function is 
E(x;J) =—5 55 Штатта 一 У; һата 
m,n 
with a separable distribution 


Q(x;a) = 2» (E auta): 


We optimize Q so as to minimize the variational free energy 


BF(a) = 8У) QGca)E(J) - У, Asa) la zc 


-8 (32 Ул — У) - Lu (dn). 


- Minimum (with respect to a) can be found by the iterative equations 
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Have you seen this before? 


Am = B (= JmnTn + із) and Zn = tanh(a,) 


or equivalently... 


Zm = tanh 区 +83) 2 
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Em = tanh (ы. +85) Ina] 


‘Mean field theory' 
is a variational method 
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